Weekly Exercises #2

library(tidyverse)     # for graphing and data cleaning

## ── Attaching packages ──────────────────────────────────────────────────────────────────────────────────── tidyverse 1.3.0 ──

## ✓ ggplot2 3.3.2     ✓ purrr   0.3.4
## ✓ tibble  3.0.3     ✓ dplyr   1.0.1
## ✓ tidyr   1.1.1     ✓ stringr 1.4.0
## ✓ readr   1.3.1     ✓ forcats 0.5.0

## ── Conflicts ─────────────────────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
## x dplyr::filter() masks stats::filter()
## x dplyr::lag()    masks stats::lag()

library(googlesheets4) # for reading googlesheet data
library(lubridate)     # for date manipulation

## 
## Attaching package: 'lubridate'

## The following objects are masked from 'package:base':
## 
##     date, intersect, setdiff, union

library(ggthemes)      # for even more plotting themes
library(babynames)     # for using the babynames dataset
library(geofacet)      # for special faceting with US map layout
gs4_deauth()           # To not have to authorize each time you knit.
theme_set(theme_minimal())       # My favorite ggplot() theme :)

#Lisa's garden data
garden_harvest <- read_sheet("https://docs.google.com/spreadsheets/d/1DekSazCzKqPS2jnGhKue7tLxRU3GVL1oxi-4bEM5IWw/edit?usp=sharing") %>% 
  mutate(date = ymd(date))

## Reading from "2020_harvest"

## Range "Sheet1"

#COVID-19 data from the New York Times
covid19 <- read_csv("https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-states.csv")

## Parsed with column specification:
## cols(
##   date = col_date(format = ""),
##   state = col_character(),
##   fips = col_character(),
##   cases = col_double(),
##   deaths = col_double()
## )

Instructions

• Put your name at the top of the document.

• For ALL graphs, you should include appropriate labels.

• Feel free to change the default theme, which I currently have set to theme_minimal().

• Use good coding practice. Read the short sections on good code with pipes and ggplot2.

• When you are finished with ALL the exercises, uncomment the options at the top so your document looks nicer. Don’t do it before then, or else you might miss some important warnings and messages.

Warm-up exercises with garden data

These exercises will reiterate the plotting and data wrangling skills you learned in the ggplot_dplyr tutorial. They will use the garden_harvest data that you are hopefully starting to become familiar with.

1. Filter the data to lettuce vegetables. Create a histogram of the weight in grams. This shows the distribution of lettuce “harvests”. There may be multiple harvests in the same day since there are multiple varieties of lettuce and I sometimes harvested lettuce more than once a day. For the lettuce, also compute the mean and standard deviation of the harvest (use the summarize() function). Use those to help you describe the distribution of lettuce harvests.

garden_harvest %>% 
  filter(vegetable == "lettuce") %>% 
  group_by(vegetable) %>% 
  summarize(mean = mean(weight), stdev = sd(weight))

## `summarise()` ungrouping output (override with `.groups` argument)

garden_harvest %>%  
  filter(vegetable == "lettuce") %>% 
  group_by(vegetable) %>% 
  ggplot(aes(x = weight)) +
  geom_histogram(fill = "light blue", color = "black") +
  labs(title = "The Weight of Lettuce Harvest",
       x = "Weight (grams)",
       y = "Total Amount")

## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.

The average weight of lettuce harvest is 76 grams, with a pretty high standard deviation of 56 grams. Most of the time the harvest weight is less than 100 grams.

2. Create a bar chart that shows the number of times I harvested the different varieties of lettuce. Put the variety on the y-axis and count on x-axis. CHALLENGE: order them so the variety I harvested the most is at the top.

garden_harvest %>% 
  filter(vegetable == "lettuce") %>% 
  group_by(vegetable, variety) %>% 
  summarize(count = n()) %>% 
  arrange(desc(count)) %>% 
  ggplot(aes(x = count, y = fct_reorder(variety, count))) +
  geom_bar(stat = "identity", fill = "darkturquoise", color = "white") + 
  labs(title = "The Harvest of Each Varieties of Lettuce",
       x = "Count",
       y = "Variety")

## `summarise()` regrouping output by 'vegetable' (override with `.groups` argument)

3. Filter the data to beets. Summarize the data to compute the total weight in grams harvests on each day for each variety of beet. Add two new variables: a. weight in pounds, b. cumulative weight in pounds. Create a line graph that shows the cumulative harvest by day for each variety of beet, with a different color for each variety. Describe what you see. CHALLENGE: make the line colors correspond with the colors of the beets - a reddish-purple color and a dark-yellow color.

garden_harvest %>% 
  filter(vegetable == "beets") %>% 
  group_by(variety, date) %>% 
  mutate(weight_lb = weight * 0.00220462,
         tot_weight_lb = sum(weight_lb),
         cum_weight_lb = cumsum(tot_weight_lb)) %>% 
  ggplot(aes(x = date, y = cum_weight_lb, color = variety)) + 
  geom_line() + 
  scale_color_manual(values=c("#D89000", "#00BA42", "#E76BF3")) +
  labs(title = "Cumulative Harvest by Day for Each Variety of Beet",
       x = "Date",
       y = "Cumulative Weight (lb)")

4. Summarize the data to compute the daily harvest in pounds for each vegetable. Create side-by-side boxplots to compare the distributions of daily harvests for each vegetable.

garden_harvest %>% 
  group_by(vegetable, date) %>% 
  mutate(weight_lb = weight * 0.00220462) %>% 
  summarise(daily_harvest_lb = sum(weight_lb)) %>% 
  ggplot(aes(x = daily_harvest_lb, y = vegetable)) +
  geom_boxplot() +
  labs(title = "Daily Harvest in Pounds for Each Vegetable",
       x = "Daily Harvest (lb)",
       y = "Vegetable")

## `summarise()` regrouping output by 'vegetable' (override with `.groups` argument)

Babynames exercises

We are going to practice more dplyr and ggplot2 skills on the babynames dataset from the babynames library. First, install the package (unless you’re using the server, in which case it is already installed). Then, in the libraries section at the top, uncomment the line where that library is loaded and re-run that code chunk. BTW, this is my kids’ favorite dataset … yes, my kids have a favorite dataset.

“Mom, can we please graph the names data?” Yes, yes we can. #rstats #futuredatascientist #data #babynames pic.twitter.com/LGgEQRCpN1

— lisa lendway (@lisalendway) September 17, 2018

A couple notes about this dataset:

1. As with many datasets that contain the variable sex, in this dataset it is binary. Hopefully this will change in the future.
   
2. Only names with at least 5 uses in a year are included.
   
3. The data come from the Social Security Administration. In the past, not everyone needed or was able to obtain a social security number. So those people are not reflected in the data.

5. Add a new variable to babynames called has2000 that indicates whether the name was used more than 2000 times within each sex and year.

babynames %>% 
  mutate(has2000 = n > 2000)

6. Add on to the code you wrote above and compute the proportion of names each year that had more than 2000 babies. Do this separately for each sex. TIP: you can use TRUEs and FALSEs mathematically. TRUE = 1 and FALSE = 0.

babynames %>% 
  mutate(has2000 = n > 2000) %>% 
  group_by(year, sex) %>% 
  summarise(above_2k_prop = mean(has2000)) # sum(has2000) / n()

## `summarise()` regrouping output by 'year' (override with `.groups` argument)

7. Create a line graph from the dataset you created in the previous problem. Year should be along the x-axis, proportion of names with more than 2000 babies is on the y-axis, and there should be a separate line for each sex.

babynames %>% 
  mutate(has2000 = n > 2000) %>% 
  group_by(year, sex) %>% 
  summarise(above_2k_prop = mean(has2000)) %>% 
  ggplot(aes(x = year, y = above_2k_prop, color = sex)) + 
  geom_line() + 
  labs(title = "Proportion of Names with More Than 2000 Babies over Years",
       x = "Year",
       y = "Proportion")

## `summarise()` regrouping output by 'year' (override with `.groups` argument)

8. Find the most popular names for males, over all years in the data, and ordered by popularity.

babynames %>% 
  filter(sex == "M") %>%
  group_by(name) %>% 
  summarise(total_n = sum(n)) %>% 
  arrange(desc(total_n))

## `summarise()` ungrouping output (override with `.groups` argument)

9. Find the most popular names for females, over all years in the data, and ordered by popularity.

babynames %>% 
  filter(sex == "F") %>% 
  group_by(name) %>% 
  summarise(total_n = sum(n)) %>% 
  arrange(desc(total_n))

## `summarise()` ungrouping output (override with `.groups` argument)

10. For each year and sex in the data, find the number of distinct/unique names. For example, if in 1889 all males were named Matthew, Christopher, or John, then there would be 3 distinct names for males in 1889 (in reality, there are more). Graph the results and describe what you see.

babynames %>% 
  group_by(year, sex) %>% 
  summarise(distinct_names = n()) %>% 
  ggplot(aes(x = year, y = distinct_names, color = sex)) + 
  geom_line() +
  labs(title = "Unique Names for Each Year")

## `summarise()` regrouping output by 'year' (override with `.groups` argument)

11. (CHALLENGE) In this exercise, we want to find out how using “popular” names has changed over time. For each year and sex, find the proportion of names that are in the top 10 names. Use this to create a line graph with a different line for each sex. What do you observe? HINT: Use the top_n() function. Take time to work through the small steps and write out what it is you need to do in words before jumping into the code.

babynames %>% 
  group_by(year, sex) %>% 
  top_n(10, wt = n) %>% 
  summarise(tot_prop = sum(prop)) %>% 
  ggplot(aes(x = year, y = tot_prop, color = sex)) +
  geom_line() + 
  labs(title = "Proportion of Names That Are In The Top 10 Names",
       y = "total prop")

## `summarise()` regrouping output by 'year' (override with `.groups` argument)

COVID-19 exercises

These exercises will use data on coronavirus cases in the US provided by the New York Times, which I have called covid19. You can (and should) read details about the data here. We will be using the state-level data, which includes the cumulative number of cases for each state for each date, starting on the date they had their first case.

In this class, I think it is important the we address and use relevant data, even if it is challenging. That being said, I do not want these data to increase your stress level. If you would prefer not to work with these data, please let me know and I will provide alternative exercises.

12. Find the date of each state’s first case. Order the states from latest to most recent first cases.

covid19 %>% 
  group_by(state) %>% 
  summarise(first_case = min(date)) %>% 
  arrange(first_case)

## `summarise()` ungrouping output (override with `.groups` argument)

13. Let’s examine data for Minnesota, Wisconsin, Iowa, North Dakota, and South Dakota. Create a line graph of cumulative case count by date and color the lines by state. What do you observe? Is this a “fair” comparison, do you think?

covid19 %>% 
  filter(state %in% c("Minnesota","Wisconsin","Iowa","North Dakota","South Dakota")) %>% 
  group_by(state, date) %>% 
  summarize(total_case = cumsum(cases)) %>% 
  ggplot(aes(x = date, y = total_case, color = state)) +
  geom_line() + 
  labs(title = "Cumulative Case Count")

## `summarise()` regrouping output by 'state' (override with `.groups` argument)

I saw a model of exponential growth for each state. Maybe it’s not a fair competition because each state has different number of population. A better way is to compare the proportion of cases among population.

14. Create the same plot as above, but make the y-axis log scale by using the scale_y_log10() function (search for it in Help or run ?scale_y_log10() in the console). What do you observe? Is this a “fair” comparison? Consider this plot and the previous one, how else might you improve them?

covid19 %>% 
  filter(state %in% c("Minnesota","Wisconsin","Iowa","North Dakota","South Dakota")) %>% 
  group_by(state, date) %>% 
  summarize(total_case = cumsum(cases)) %>% 
  ggplot(aes(x = date, y = total_case, color = state)) +
  geom_line() + 
  scale_y_log10() +
  labs(title = "Cumulative Case Count")

## `summarise()` regrouping output by 'state' (override with `.groups` argument)

On the logged y axis, moving one unit is like multiplying by ten. This second graph also reflects the rate of change.

15. In this exercise we are going to compute some new variables and get set up for the next exercise. For each state, compute a 1-day lag and 7-day lag of cumulative counts (use the lag() function). When you do this, there will be some missing values because at the beginning there are no lags. Use the replace_na() function to replace those missing values with 0’s. Add two new variables: a. a variable that computes the number of new cases for each day, which can be computed by taking the current cases minus the 1-day lag. b. a variable that computes the 7 day average, which can be computed by taking the current cases minus the 7-day lag and dividing that by 7.

covid_lag <-
  covid19 %>% 
    group_by(state) %>% 
    mutate(one_day_lag = lag(cases, 1), 
           seven_day_lag = lag(cases, 7),
           one_day_lag = replace_na(one_day_lag, 0),
           seven_day_lag = replace_na(seven_day_lag, 0),
           newcases_day = cases - one_day_lag,
           seven_day_avg = (cases - seven_day_lag) / 7)

covid_lag

16. Using the data you created in the previous step (you can %>% that data into ggplot()), create a plot with the following: a. facet by state, b. a line with the number of new cases by date, b. a line with the 7-day average number of new cases by date - make this line blue. What do you observe? How could this graph be improved?

covid_lag %>% 
  ggplot(aes(x = date)) + 
  geom_line(aes(y = newcases_day), color = "red") + 
  geom_line(aes(y = seven_day_avg), color = "blue") +
  facet_wrap(vars(state)) +
  labs(title = "Number of New Cases and 7 Day Avg Number of New Cases by Date")

I find in most state there are just 2 straight lines. This is because while some state has much more population, the y axis for all the states are the same, which should be improved.

17. For this part, you will need the geofacet library. First, install it (unless you are using the server, in which case it will already be there for you). Then, uncomment the library statement in the 2nd R code chunk above and re-run that R code chunk. Create the same plot as above but instead of regular faceting, use facet_geo(). Also set scales="free" to the facet function. What do you learn from this plot that you didn’t learn from the previous one?

covid_lag %>% 
  ggplot(aes(x = date)) + 
  geom_line(aes(y = newcases_day), color = "red") + 
  geom_line(aes(y = seven_day_avg), color = "blue") +
  facet_geo(vars(state), scales = "free") +
  labs(title = "Number of New Cases and 7 Day Avg Number of New Cases by Date")

## Warning: Using `as.character()` on a quosure is deprecated as of rlang 0.3.0.
## Please use `as_label()` or `as_name()` instead.
## This warning is displayed once per session.

## Some values in the specified facet_geo column 'state' do not match the
##   'name' column of the specified grid and will be removed: Puerto Rico,
##   Virgin Islands, Guam, Northern Mariana Islands

I could also see the geography of each data point in this second graph.